Rendering device and rendering method

ABSTRACT

A rendering device that renders a three-dimensional object displayed in a virtual reality space on a display includes a processor and a memory storing instructions that, when executed by the processor, cause the processor to render the 3D object as a 3D object of 3D displaying in a virtual reality space coordinate system, render the 3D object as a 3D object of 2D displaying in a plane coordinate system, and update displaying of the display based on a result of the 3D object being rendered as the 3D object of 3D displaying in the virtual reality space coordinate system and a result of the object being rendered as the 3D object of 2D displaying in the plane coordinate system. The rendering device enables intuitive drawing with high accuracy in the virtual reality space.

BACKGROUND Technical Field

The present disclosure relates to a rendering device that carries outrendering of a 3D (Three-Dimensional) object in a virtual reality(including VR: Virtual Reality, AR: Augmented Reality, MR: MixedReality) space and a rendering method.

Background Art

In recent years, needs to design various products while carrying outstereoscopic viewing in a virtual reality space have been increasing.

Patent Document 1 discloses a technique in which a 3D object isgenerated based on a 2D (Two-Dimensional) object input to a tabletterminal by using an electronic pen in an AR space. Hereinafter, themethod of input carried out by moving an electronic pen on a plane willbe referred to as “2D input.”

Non Patent Document 1 discloses a technique in which a 3D object isinput by moving a controller for 3D input in a VR space. However, themethod of input with use of such a controller for 3D input will bereferred to as “3D input.”

PRIOR ART DOCUMENT

Patent Document

-   Patent Document 1: U.S. Patent Application Publication No.    2016/0343174

Non Patent Document

-   Non Patent Document 1: Google, “Tilt Brush: Painting from a new    perspective,” [online], May 3, 2016 [Retrieved on Oct. 5, 2017], the    Internet <URL: https://www.youtube.com/watch?v=TckqNdrdbgk>

BRIEF SUMMARY Technical Problems

According to the above-described 2D input, the position of theelectronic pen is fixed in a known plane and therefore high accuracy canbe obtained compared with the 3D input. However, meanwhile, in the 2Dinput, there is a problem that intuitive manipulation is difficultcompared with the 3D input because the 2D input is limited to drawing inthe plane.

In contrast to this, according to the 3D input, intuitive manipulationbecomes possible. Meanwhile, there is a problem that the accuracy isinsufficient as a technique used for the purpose of design because theflexibility in the position of the controller is high.

Therefore, one of objects of the present disclosure is to provide arendering device and a rendering method that can implement intuitivedrawing with high accuracy in a virtual reality space.

Technical Solution

A first aspect of the present disclosure relates to a rendering devicethat renders a 3D object in a virtual reality space displayed on avirtual reality display. The rendering device includes a processor; anda memory storing instructions that, when executed by the processor,cause the processor to: render the 3D object as a 3D object of 3Ddisplaying in a virtual reality space coordinate system, render the 3Dobject as a 3D object of 2D displaying in a plane coordinate system, andupdate displaying of the virtual reality display based on a result ofthe 3D object being rendered as the three-dimensional object ofthree-dimensional displaying in the virtual reality space coordinatesystem and displaying of the virtual reality display based on a resultof the 3D object being rendered as the 3D object of 2D displaying in theplane coordinate system.

A second aspect of the present disclosure relates to a rendering methodfor rendering a 3D object in a virtual reality space displayed on avirtual reality display. The rendering method includes rendering the 3Dobject as a 3D object of 3D displaying in a virtual reality spacecoordinate system, rendering the 3D object as a 3D object of 2Ddisplaying in a plane coordinate system, and updating displaying of thevirtual reality display based on a result of the rendering the 3D objectas the three-dimensional object of three-dimensional displaying in thevirtual reality space coordinate system and the displaying of thevirtual reality display based on a result of the rendering 3D object asthe 3D object of 2D displaying in the plane coordinate system.

Advantageous Effect

According to the present disclosure, the display method (3D displayingor 2D displaying) of the 3D object in the virtual reality space can beswitched according to selection by the user and therefore it becomespossible to implement intuitive drawing with high accuracy in thevirtual reality space.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating a configuration of a 3D objectrendering system 1 according to a first embodiment of the presentdisclosure.

FIG. 2 is a diagram explaining a relationship between a tablet surfacecoordinate system and a virtual reality space coordinate system.

FIG. 3 is a flowchart illustrating processing performed by a processor 2a illustrated in FIG. 1.

FIG. 4 is a flowchart illustrating details of acquisition processing ofposition information illustrated in FIG. 2.

FIG. 5 is a flowchart illustrating details of tablet terminal displayprocessing illustrated in FIG. 2.

FIG. 6 is a diagram illustrating a state in which 2D input of a 3Dobject is being carried out in a virtual reality space according to thefirst embodiment of the present disclosure.

FIG. 7 is a diagram illustrating a state in which 3D input of a 3Dobject is being carried out in a virtual reality space according to thefirst embodiment of the present disclosure.

FIG. 8 is a diagram illustrating a state in which 2D input of a 3Dobject is being carried out in a virtual reality space according to asecond embodiment of the present disclosure.

FIGS. 9A and 9B are flowcharts illustrating part of processing performedby the processor 2 a according to the second embodiment of the presentdisclosure.

MODES FOR CARRYING OUT THE DISCLOSURE

Embodiments of the present disclosure will be described in detail belowwith reference to the accompanying drawings.

FIG. 1 is a diagram illustrating a configuration of a 3D objectrendering system 1 according to a first embodiment of the presentdisclosure. As illustrated in this diagram, the 3D object renderingsystem 1 according to the present embodiment is configured to include acomputer 2, a virtual reality display 3, a tablet 4, an electronic pen5, a glove unit 6, lightning houses 7 a and 7 b, and position sensors 8a to 8 d. The position sensors 8 a, 8 c, and 8 d are attached to thetablet 4, the electronic pen 5, and the glove unit 6, respectively, andthe position sensor 8 b is disposed in the virtual reality display 3.Due to the attaching of the position sensor 8 c, the electronic pen 5functions as a stylus and functions also as a controller. The positionsensor 8 c may be incorporated in the electronic pen 5.

Each device illustrated in FIG. 1 is disposed in a room in principle. Inthe 3D object rendering system 1, almost the whole room can be used as avirtual reality space.

The computer 2 includes a processor 2 a (controller) and a memory 2 bthat cooperates with the processor 2 a. Each kind of processingperformed by the computer 2 to be described later can be implementedthrough cooperation between the processor 2 a and the memory 2 b (morespecifically, through reading-out and execution of a program stored inthe memory 2 b). In other words, the memory 2 b stores instructionsthat, when executed by the processor 2 a, causes the computer 2 toperform the various kinds of processing described herein.

The computer 2 is connected to each of the virtual reality display 3 andthe lightning houses 7 a and 7 b in a wired or wireless manner. In FIG.1, an example in which the computer 2 is connected to each of thevirtual reality display 3 and the lightning houses 7 a and 7 b based ona wired communication standard such as USB (Universal Serial Bus) isillustrated. Furthermore, as described in detail later, the computer 2is connected also to the tablet 4 in a wired or wireless manner if thetablet 4 has a communication function. In FIG. 1, an example in whichthe computer 2 and the tablet 4 are connected based on a short distancewireless communication standard such as Bluetooth (registeredtrademark), for example, is illustrated. If the tablet 4 or the virtualreality display 3 includes functions as a computer, the computer 2 maybe configured by the computer.

The computer 2 is configured to have a function of displaying a virtualreality space on the virtual reality display 3. This virtual realityspace may be a VR (Virtual Reality) space or may be an AR (AugmentedReality) space or may be an MR (Mixed Reality) space. In the case ofdisplaying a VR space, the user who wears the virtual reality display 3recognizes virtual reality and is separated from the actual world. Onthe other hand, in the case of displaying an AR space or MR space, theuser who wears the virtual reality display 3 recognizes a space in whichvirtual reality and the actual world are mixed.

The computer 2 is configured to function as a rendering device that setsa virtual reality space based on the positions of the lightning houses 7a and 7 b and carries out rendering of various 3D objects in the setvirtual reality space. The computer 2 updates displaying of the virtualreality display 3 based on the result of the rendering. Due to this,various 3D objects appear in the virtual reality space displayed on thevirtual reality display 3.

The rendering by the computer 2 is carried out based on a 3D objectstored in the memory 2 b. The 3D object is information that representsthe shape, position, and orientation of a 3D object in a virtual realityspace coordinate system that represents the virtual reality space set bythe computer 2, and is stored in the memory 2 b regarding each 3D objectof a rendering target.

In the 3D objects rendered by the computer 2, 3D objects that representeach of the tablet 4, the electronic pen 5, and the glove unit 6illustrated in FIG. 1 are included. In rendering of these 3D objects,first the computer 2 detects the position and orientation of each of theposition sensors 8 a to 8 d in the virtual reality space coordinatesystem. Furthermore, the computer 2 is configured to acquirepoint-of-view information that represents the point of view of the userbased on the detected position and orientation of the position sensor 8b and carry out rendering of the 3D objects that represent each of thetablet 4, the electronic pen 5, and the glove unit 6 in the virtualreality space based on the acquired point-of-view information, the shapeof each 3D object stored, and the detected position and orientation ofeach of the position sensors 8 a, 8 c, and 8 d.

Moreover, the computer 2 is configured to be capable of detectingmanipulation carried out by the user in the virtual reality space bydetecting the position of the position sensors 8 c and 8 d and newlycreating a 3D object or updating a 3D object that has been already heldbased on the result thereof.

The virtual reality display 3 is a VR display mounted on the head of ahuman to be used (head-mounted display). As virtual reality displaysthat are generally commercially available, there are various kinds ofdisplays such as “transmissive type” or “non-transmissive type” and“glasses type” or “hat type.” Any can be used as the virtual realitydisplay 3.

The virtual reality display 3 is connected to each of the positionsensor 8 a, the electronic pen 5 (including position sensor 8 c), andthe glove unit 6 (including position sensor 8 d) in a wired or wirelessmanner. The position sensors 8 a, 8 c, and 8 d are configured to notifylight reception level information to be described later to the virtualreality display 3 through this connection. The virtual reality display 3is configured to notify the light reception level information notifiedfrom each of the position sensors 8 a, 8 c, and 8 d to the computer 2with the light reception level information of the position sensor 8 bincorporated in the virtual reality display 3. The computer 2 detectsthe position and orientation of each of the position sensors 8 a to 8 din the virtual reality space coordinate system based on the lightreception level information thus notified. Furthermore, the electronicpen 5 and the glove unit 6 are configured to notify manipulationinformation to be described later to the virtual reality display 3through the above-described connection. The virtual reality display 3 isconfigured to transfer the manipulation information thus notified to thecomputer 2.

The tablet 4 has a tablet surface 4 a. It is preferable for the tabletsurface 4 a to be a flat surface, and the tablet surface 4 a can beformed of a material suitable to slide the pen tip of the electronic pen5. In one example, the tablet 4 is what is called a digitizer and isconfigured to have a touch sensor that detects the indicated position bythe electronic pen 5 in a touch surface and a communication function ofnotifying the detected indicated position to the computer 2. The tabletsurface 4 a in this case is formed of the touch surface of thedigitizer. In another example, the tablet 4 is what is called a tabletcomputer and is configure to have a display, a touch sensor that detectsthe indicated position by the electronic pen 5 in a display surface ofthis display, and a communication function of notifying the detectedindicated position to the computer 2. The tablet surface 4 a in thiscase is formed of the display surface of the display. In further anotherexample, the tablet 4 is a physical object that does not have a functionof detecting the indicated position by the electronic pen 5 (includingmere plate, table, display or computer that does not have a function ofdetecting the indicated position by the electronic pen 5,). The tabletsurface 4 a in this case is formed of a flat surface made in the surfaceof the tablet 4.

The position sensors 8 a are fixedly set on the surface of the tablet 4.Therefore, the position and orientation of the position sensors 8 adetected by the computer 2 represent the position and orientation of thetablet surface 4 a in the virtual reality space coordinate system.

The electronic pen 5 and the glove unit 6 enable the user to indicatethe position in the virtual reality space. The electronic pen 5 isconfigured to have a pen shape. The glove unit 6 has a shape as a gloveworn on a hand of the user.

Various sensing devices such as a switch are disposed on the surface ofthe electronic pen 5 or inside the electronic pen 5. The term sensingdevice here includes a sensor configured to be capable of detecting anyphysical quantity besides a changeover switch that takes either theon-state or the off-state. As an example of the switch disposed for theelectronic pen 5, a side switch or tail switch configured to be capableof accepting on/off-manipulation by the user is cited. Furthermore, asan example of another switch disposed for the electronic pen 5, acapacitance sensor that detects the pressure (writing pressure) appliedto the pen tip of the electronic pen 5 are also cited. The electronicpen 5 is configured to detect the output of the switch disposed for theelectronic pen 5 (pressed-down state or detected physical quantity) andnotify all or part of the detection result to the virtual realitydisplay 3 as its own manipulation information.

As the sensing device disposed for the electronic pen 5, a force sensor(load sensor) that detects the gripping force of the electronic pen 5 bythe user may be included. In this case, although the output of the forcesensor is not what represents the writing pressure originally, thecomputer 2 that has received notification of the output of the forcesensor may treat it as data that represents the writing pressure(writing pressure data). This makes it possible to reflect the writingpressure in the drawing result even when the user manipulates theelectronic pen 5 in the air.

If the tablet 4 has a touch sensor, the position of the electronic pen 5is detected also by this touch sensor. The position detected by thetouch sensor is not a position in the virtual reality space coordinatesystem but a position in a tablet surface coordinate system defined onthe tablet surface 4 a. The touch sensor is configured to notify thedetected position of the electronic pen 5 to the computer 2.

In general, the position detected by the touch sensor has higheraccuracy than the position detected by using the position sensor 8 c.Therefore, it is preferable for the computer 2 to acquire not a positiondetected through the position sensor 8 c but a position notified fromthe touch sensor as the position of the electronic pen 5 when theposition of the electronic pen 5 has been notified from the touchsensor. In this case, it is preferable for the computer 2 to convert theposition notified from the touch sensor to a position in the virtualreality space coordinate system by associating the tablet surfacecoordinate system with the virtual reality space coordinate system basedon the position and orientation of the tablet surface 4 a detected byusing the position sensors 8 a.

FIG. 2 is a diagram explaining the relationship between the tabletsurface coordinate system and the virtual reality space coordinatesystem. In this diagram, the state in which the tablet surface 4 a islocated in a virtual reality space 10 is illustrated. The virtualreality space coordinate system is defined by three axes VRX, VRY, andVRZ and the tablet surface coordinate system is defined by three axesTRX, TRY, and TRZ. The axis TRZ is the normal direction of the tabletsurface 4 a. When a position P illustrated in the diagram is detected asa position (x, y, z) in the tablet surface coordinate system by thetouch sensor (z represents the hover position, for example), thecomputer 2 converts this (x, y, z) to a position (X, Y, Z) in thevirtual reality space coordinate system by predetermined conversionprocessing. This makes it possible to convert the position notified fromthe touch sensor to the position in the virtual reality space coordinatesystem.

Here, for the position detection of the electronic pen 5 by the touchsensor, the electromagnetic induction system may be used or the activecapacitive system may be used. When the active capacitive system isused, the touch sensor is configured to send out a beacon signal from asensor electrode (not illustrated) disposed in the touch surface atpredetermined time intervals. In the beacon signal, a command forcontrolling the electronic pen 5 from the touch sensor is included. Inthe contents of control based on the command, for example, transmissionof writing pressure data (what has been detected by a capacitancesensor) that represents the pressure applied to the pen tip of theelectronic pen 5, transmission of the pressed-down state of variousswitches (not illustrated) disposed for the electronic pen 5,transmission of a unique ID (Identification) stored in the electronicpen 5 in advance, are included.

When detecting the above-described beacon signal, the electronic pen 5compatible with the active capacitive system sends out a pen signal as aresponse signal. The pen signal is a signal including a burst signalthat is an unmodulated carrier wave and a data signal obtained bymodulating a carrier wave with data according to the above-describedcommand. The touch sensor attempts detection of the burst signal by theabove-described sensor electrode and detects the position of theelectronic pen 5 based on the detection result. Furthermore, the touchsensor receives data transmitted by the electronic pen 5 according tothe command by detecting the data signal by the above-described sensorelectrode and demodulating the data signal. The tablet 4 is configuredto transmit the position of the electronic pen 5 and the datatransmitted by the electronic pen 5, acquired in this manner, to thecomputer 2. The computer 2 is configured to convert the position thusnotified to the position in the virtual reality space coordinate systemin the above-described manner and acquire the notified data as part ofthe above-described manipulation information.

The lightning houses 7 a and 7 b are signal transmitting devices forposition detection used in the 3D object rendering system 1 and are eachconfigured to be capable of emitting a signal, laser light in thisexample, while changing the direction in accordance with control by thecomputer 2. The position sensors 8 a to 8 d are each composed of plurallight receiving sensors and are configured to receive the signal (laserlight) applied by each of the lightning houses 7 a and 7 b by therespective light receiving sensors and acquire light reception levelinformation including the respective light reception levels. Theacquired light reception level information is notified from therespective position sensors 8 a to 8 d to the computer 2 and is used fordetecting the position and orientation of them as described above.

The overall outline of the 3D object rendering system 1 is describedabove. When a user inputs a new 3D object in such a 3D object renderingsystem 1, until now, either of the following operations is carried out:3D input is carried out by using the electronic pen 5 and the glove unit6; and a tablet computer is employed as the tablet 4 and 2D input iscarried out to this tablet computer by using the electronic pen 5.However, as described above, the 3D input involves a disadvantage thatthe accuracy is insufficient while having an advantage that intuitivemanipulation becomes possible. Furthermore, the 2D input involves adisadvantage that intuitive manipulation is difficult whereas highaccuracy can be obtained.

In view of problems of such conventional input methods, the 3D objectrendering system 1 according to the present embodiment enablesimplementation of intuitive drawing with high accuracy in a virtualreality space by allowing the display method (3D displaying or 2Ddisplaying) of a 3D object in the virtual reality space to be switchedby selection by the user. This point will be described in detail belowwith reference to a flowchart of processing performed by the processor 2a of the computer 2.

FIG. 3 is the flowchart illustrating the processing performed by theprocessor 2 a of the computer 2. This processing is performed when auser inputs a 3D object by using at least one of the electronic pen 5and the glove unit 6, and is started through performance ofpredetermined manipulation by the user with use of the electronic pen 5and the glove unit 6 and detection of it by the processor 2 a.

As illustrated in FIG. 3, first the processor 2 a stores, in the memory2 b, data of a 3D object that represents the shape, position, andorientation of a currently-input object in a virtual reality spacecoordinate system (S1). Although the specific format of the 3D object isnot particularly limited, it is preferable to employ data of a VRML(Virtual Reality Modeling Language) format or X3D (eXtensible 3D)format, for example.

Next, the processor 2 a performs acquisition processing of positioninformation (S2).

FIG. 4 is a flowchart illustrating details of the acquisition processingof position information performed at S2. As illustrated in this diagram,the processor 2 a performs the acquisition processing of positioninformation by acquiring point-of-view information (S20), acquiringtablet surface information (S21), acquiring controller information(first information) (S22, controller information acquisition, firstinformation acquisition processing), and acquiring electronic peninformation (second information) (S23, electronic pen informationacquisition, second information acquisition processing). The order ofperformance of S20 to S23 is not particularly limited.

The point-of-view information is information that represents the pointof view of the user in the virtual reality space coordinate system andspecifically is represented by the position and orientation of thevirtual reality display 3. The processor 2 a is configured to acquirethe point-of-view information based on the position and the orientationdetected regarding the position sensor 8 b. The specific point-of-viewinformation is composed of vector information having onethree-dimensional coordinate as the starting point, for example.

The tablet surface information is information that represents the shape,position, and orientation of the tablet surface 4 a in the virtualreality space coordinate system and is stored in the memory 2 b as oneof 3D objects. The processor 2 a acquires the tablet surface informationbased on the position and the orientation detected regarding theposition sensors 8 a and the shape of the tablet 4 stored in advance.

The controller information is information that represents the positionand orientation of the 3D controller (including electronic pen 5 andglove unit 6) in the virtual reality space coordinate system andmanipulation information of this 3D controller. The processor 2 aacquires the controller information based on the position and theorientation detected regarding the position sensors 8 c and 8 d andmanipulation information of each of the electronic pen 5 and the gloveunit 6 received through the virtual reality display 3.

The electronic pen information is information that represents theindicated position by the electronic pen 5 in the tablet surfacecoordinate system and manipulation information of the electronic pen 5.When the tablet 4 has a touch sensor, the processor 2 a acquires theindicated position by the electronic pen 5 in the tablet surfacecoordinate system from the touch sensor. On the other hand, when thetablet 4 does not have the touch sensor, the processor 2 a acquires theindicated position by the electronic pen 5 in the tablet surfacecoordinate system by performing conversion processing (reverseprocessing of conversion processing described with reference to FIG. 2)for the position acquired from the position sensor 8 c (position in thevirtual reality space coordinate system). Furthermore, when theprocessor 2 a can acquire the manipulation information of the electronicpen 5 from the touch sensor (for example, when the electronic pen 5 iscompatible with the active capacitive system), the processor 2 aacquires the manipulation information of the electronic pen 5 (includingoutput of the capacitance sensor) from the touch sensor. On the otherhand, when the processor 2 a cannot acquire the manipulation informationof the electronic pen 5 from the touch sensor, the processor 2 aacquires the manipulation information of the electronic pen 5 (includingoutput of the force sensor) through the virtual reality display 3.

Referring back to FIG. 3, the processor 2 a that has performed theacquisition processing of position information subsequently performsdisplay processing of tablet surface (S3). This processing is processingfor displaying, in the virtual reality space, a tablet surface imagethat illustrates the tablet surface and a display surface image thatillustrates a display surface for 2D displaying of a currently-input 3Dobject.

FIG. 5 is a flowchart illustrating details of the display processing oftablet surface performed in the S3. As illustrated in this diagram,first the processor 2 a carries out rendering of the display surfaceimage based on a position selected by the user in the virtual realityspace or a position in contact with the currently-input 3D object (S30,display surface image rendering (processing)). The selection of theposition by the user is carried out through pressing-down of a switchdisposed for the electronic pen 5 by the user and notification ofmanipulation information representing this to the processor 2 a, forexample. The display surface image may be what imitates a display or maybe a mere rectangular frame. It is preferable for the processor 2 a tocarry out rendering of the display surface image in such a manner thatthe normal direction of the display surface image corresponds with thedirection of the line of sight of the user based on the point-of-viewinformation acquired at S20 in FIG. 4.

Next, the processor 2 a acquires first correspondence information thatrepresents the correspondence relationship between the virtual realityspace coordinate system and a display surface coordinate system definedon the display surface (first plane coordinate system) (S31).Specifically, the first correspondence information is a conversion rulefor mutually converting the virtual reality space coordinate system andthe display surface coordinate system.

Subsequently, the processor 2 a acquires second correspondenceinformation that represents the correspondence relationship between thevirtual reality space coordinate system and the tablet surfacecoordinate system (second plane coordinate system) (S32). Specifically,the second correspondence information is a conversion rule for mutuallyconverting the virtual reality space coordinate system and the tabletsurface coordinate system.

At last, the processor 2 a carries out rendering of the tablet surfaceimage that represents the tablet surface 4 a based on the tablet surfaceinformation and the point-of-view information (S33, tablet surface imagerendering (processing)). The tablet surface image may be what imitates atouch surface of a tablet terminal or may be a mere rectangular frame.It is preferable for the processor 2 a to carry out rendering of thetablet surface image in such a manner that the angle formed by thedirection of the line of sight of the user and the tablet surface 4 a(angle of depression) becomes a predetermined value based on thepoint-of-view information acquired at S20 in FIG. 4.

Referring back to FIG. 3, the processor 2 a that has performed thedisplay processing of tablet surface subsequently accepts selectionmanipulation of the operation mode by the user (S4). For example, thisselection may be carried out through pressing-down of a switch disposedfor the electronic pen 5 by the user or may be carried out throughchanging the distance between the tablet surface 4 a and the electronicpen 5 by the user. For example, switching to the 2D displaying may becarried out when the distance between the tablet surface 4 a and theelectronic pen 5 has become shorter than a predetermined distance, andswitching to the 3D displaying may be automatically carried out when thedistance between the tablet surface 4 a and the electronic pen 5 hasbecome equal to or longer than the predetermined distance. In the lattercase, the distance between the tablet surface 4 a and the electronic pen5 may be detected by the touch sensor or may be detected by theprocessor 2 a based on the display position of the tablet surface imageand the position of the electronic pen 5 that has been detected by usingthe position sensor 8 c. The processor 2 a that has accepted theselection manipulation at S4 performs processing of making an entry intoeither one of the 3D display mode or the 2D display mode according tothe contents of the selection (that is, display selection processing ofselecting either one of 3D displaying or 2D displaying) (S5, modeselection).

When an entry into the 3D display mode is made at S5, the processor 2 acarries out a 3D rendering act (processing) of rendering a 3D object inthe virtual reality space coordinate system on the virtual realitydisplay 3. Specifically, first the processor 2 a carries out renderingof the currently-input object and other 3D objects based on the 3Dobject stored in the memory 2 b and the point-of-view informationacquired at S20 in FIG. 4 (S6). Then, the processor 2 a updates theoutput to the virtual reality display 3 (displaying) based on the resultthereof (S7, display update (processing)). At this time, other displaysin the virtual reality space, such as the display surface image renderedat S30 in FIG. 5 and the tablet surface image rendered at S33 in FIG. 5,are also simultaneously updated. This enables the user to edit thecurrently-input object by the 3D input.

Moreover, the processor 2 a updates the 3D object of the currently-inputobject stored in the memory 2 b based on the controller informationacquired at S22 in FIG. 4 (S8, 3D object update (processing)).Thereafter, the processor 2 a returns to the S2 to continue theprocessing.

Here, the update of the 3D object at S8 and S14 to be described later iscarried out also based on the manipulation information that has beennotified from the electronic pen 5. For example, when data thatrepresents the output of the capacitance sensor or the output of theforce sensor is notified from the electronic pen 5, the computer 2acquires this data as writing pressure data and decides the line widthand transparency of the currently-input object based on the acquiredwriting pressure data. Then, the computer 2 reflects the decision resultin the 3D object.

The processor 2 a when an entry into the 2D display mode is made at S5carries out a 2D rendering act (processing) of rendering a 3D object onthe virtual reality display 3 as a 3D object of the 2D displaying in thedisplay surface coordinate system. Specifically, first, the processor 2a converts the 3D object of the currently-input object stored in thememory 2 b to a 2D object that represents the shape, position, andorientation of the currently-input object in the display surfacecoordinate system based on the first correspondence information acquiredat S31 in FIG. 5 (S10, first conversion). Then, the processor 2 acarries out rendering of the currently-input object based on theobtained 2D object and the point-of-view information acquired at S20 inFIG. 4 (S11) and updates the output to the virtual reality display 3(displaying) based on the result thereof (S12, display update(processing)). Also at this time, similarly to S7, other displays in thevirtual reality space, such as the display surface image rendered at S30in FIG. 5 and the tablet surface image rendered at S33 in FIG. 5, aresimultaneously updated. This enables the user to edit thecurrently-input object by the 2D input.

Subsequently, the processor 2 a converts the indicated positionrepresented by the electronic pen information acquired at S23 in FIG. 4to a position in the virtual reality space coordinate system based onthe second correspondence information acquired at S32 in FIG. 5 (S13,second conversion). Then, the processor 2 a updates the 3D object of thecurrently-input object stored in the memory 2 b based on the electronicpen information including the obtained indicated position (S14, 3Dobject update). As described above, this update is carried out alsobased on the manipulation information that has been notified from theelectronic pen 5. Thereafter, the processor 2 a returns to S2 andcontinues the processing.

FIG. 6 is a diagram illustrating a state in which 2D input of a 3Dobject is being carried out in a virtual reality space (state in which auser has selected the 2D display mode at S4 in FIG. 3). FIG. 7 is adiagram illustrating a state in which 3D input of a 3D object is beingcarried out in a virtual reality space (state in which a user hasselected the 3D display mode at S4 in FIG. 3).

As illustrated in FIG. 6, in the 2D display mode, 2D displaying of acurrently-input object 13 is carried out in a rectangular displaysurface image 11 displayed in a virtual reality space 10. Three axesDRX, DRY, and DRZ illustrated in the display surface image 11 representthe display surface coordinate system. The axis DRZ in the three axes isthe normal direction of the display surface image 11. Furthermore, arectangular tablet surface image 12 is displayed and the user edits thecurrently-input object 13 by moving the electronic pen 5 in the tabletsurface image 12. This editing is carried out by moving the electronicpen 5 on the plane and therefore is 2D input. Although being invisiblefor the user who wears the virtual reality display 3, the tablet surface4 a actually exists at the position at which the tablet surface image 12is displayed. Therefore, the user can move the electronic pen 5 in thetablet surface image 12 while feeling touch of the tablet surface 4 a.

On the other hand, as illustrated in FIG. 7, in the 3D display mode, 3Ddisplaying of the currently-input object 13 is carried out in thevirtual reality space 10. The user edits the currently-input object 13by moving the electronic pen 5 and the glove unit 6 in the virtualreality space 10. This editing is carried out by using the controllerfor 3D input and therefore is 3D input.

As described above, according to the 3D object rendering system 1 inaccordance with the present embodiment, the display method (3Ddisplaying or 2D displaying) of the 3D object in the virtual realityspace can be switched by selection by the user and it becomes possibleto implement intuitive drawing with high accuracy in the virtual realityspace. Furthermore, editing of the 3D object by 2D input is enabledwhile the 2D displaying is carried out, and editing of the 3D object by3D input is enabled while the 3D displaying is carried out. Therefore,it becomes possible to edit the 3D object by the input method suitablefor the display method.

Furthermore, according to the 3D object rendering system 1 in accordancewith the present embodiment, it becomes possible to input the 3D objectby the electronic pen 5 in the virtual reality space.

Next, a 3D object rendering system 1 according to a second embodiment ofthe present disclosure will be described. In that the display surfacecoordinate system and the tablet surface coordinate system are set tothe same coordinate system, the 3D object rendering system 1 accordingto the present embodiment is different from the 3D object renderingsystem 1 according to the first embodiment, in which they are coordinatesystems different from each other. The 3D object rendering system 1according to the present embodiment is the same as the 3D objectrendering system 1 according to the first embodiment in the otherpoints. Therefore, the same configuration as the first embodiment isgiven the same symbol and description will be made below with focus onthe difference from the first embodiment.

FIG. 8 is a diagram illustrating a state in which 2D input of a 3Dobject is being carried out in a virtual reality space according to thepresent embodiment. First, the outline of the present embodiment will bedescribed with reference to this FIG. 8. The processor 2 a according tothe present embodiment is configured to carry out rendering of a tabletterminal image 14 that illustrates the tablet 4 (tablet terminal)illustrated in FIG. 1 in the virtual reality space. A tablet surfaceimage 14 a included in the tablet terminal image 14 corresponds to boththe tablet surface image and the display surface image described in thefirst embodiment. Therefore, the tablet surface coordinate system (axesTRX, TRY, and TRZ) doubles as the display surface coordinate system inthe present embodiment. As a result, in the 2D display mode, 2Ddisplaying of the currently-input object 13 is carried out in the tabletsurface image 14 a as illustrated in FIG. 8.

FIGS. 9A and 9B are flowcharts illustrating part of processing performedby the processor 2 a according to the present embodiment. FIG. 9A iswhat replaces the flowchart illustrated in FIG. 5. Meanwhile, FIG. 9B iswhat replaces S10 illustrated in FIG. 3.

As illustrated in FIG. 9A, the processor 2 a according to the presentembodiment is configured not to carry out the S30 and S31 illustrated inFIG. 5 and to carry out S33 a instead of S33. At S33 a, the processor 2a carries out rendering of the tablet terminal image that illustratesthe tablet 4 (tablet terminal) illustrated in FIG. 1 based on the tabletsurface information and the point-of-view information (tablet terminalimage rendering (processing)). Due to this, as illustrated in FIG. 8,the tablet terminal (tablet terminal image 14) similar to the actualtablet 4 (tablet terminal) appears in the virtual reality space 10.

Furthermore, as illustrated in FIG. 9B, the processor 2 a according tothe present embodiment performs processing of converting the 3D objectof the currently-input object stored in the memory 2 b to a 2D objectthat represents the shape, position, and orientation of thecurrently-input object in the display surface coordinate system based onnot the first correspondence information but the second correspondenceinformation acquired at S32 in order to obtain 2D object informationused in rendering of the currently-input object in the 2D display mode(S10 a). As a result, as illustrated in FIG. 8, the currently-inputobject 13 (that is, position of the electronic pen 5 in the virtualreality space coordinate system acquired by the controller informationacquisition (S22)) is displayed in the tablet surface image 14 a.

Also by the 3D object rendering system 1 according to the presentembodiment, the display method (3D displaying or 2D displaying) of the3D object in the virtual reality space can be switched by selection bythe user. Therefore, it becomes possible to implement intuitive drawingwith high accuracy in the virtual reality space. Furthermore, editing ofthe 3D object by 2D input is enabled while the 2D displaying is carriedout, and editing of the 3D object by 3D input is enabled while the 3Ddisplaying is carried out. Therefore, it becomes possible to edit the 3Dobject by the input method suitable for the display method.

In addition, according to the present embodiment, the user is enabled toobtain an experience of input to the tablet terminal in the virtualreality space. Therefore, it becomes possible to carry out inputmanipulation of the 3D object as if input manipulation were carried outwith a normal tablet terminal.

The preferred embodiments of the present disclosure are described above.However, the present disclosure is not limited to these embodiments atall and the present disclosure can be carried out in various modes insuch a range as not to depart from the gist thereof.

DESCRIPTION OF REFERENCE SYMBOLS

-   -   1 3D object rendering system    -   2 Computer    -   2 a Processor    -   2 b Memory    -   3 Virtual reality display    -   4 Tablet    -   4 a Tablet surface    -   5 Electronic pen    -   6 Glove unit    -   7 a, 7 b Lightning house    -   8 a to 8 d Position sensor    -   10 Virtual reality space    -   11 Display surface    -   12 Tablet surface image    -   13 Currently-input object    -   14 Tablet terminal image    -   14 a Tablet surface image

1. A rendering device that renders a three-dimensional object displayedin a virtual reality space on a display, the rendering devicecomprising: a processor; and a memory storing instructions that, whenexecuted by the processor, cause the processor to: render thethree-dimensional object as a three-dimensional object ofthree-dimensional displaying in a virtual reality space coordinatesystem, render the three-dimensional object as a three-dimensionalobject of two-dimensional displaying in a plane coordinate system, andupdate displaying of the display based on a result of thethree-dimensional object being rendered as the three-dimensional objectof three-dimensional displaying in the virtual reality space coordinatesystem and displaying of the display based on a result of thethree-dimensional object being rendered as the three-dimensional objectof two-dimensional displaying in the plane coordinate system.
 2. Therendering device according to claim 1, wherein the instructions, whenexecuted by the processor, cause the processor to: acquire firstinformation indicating a position of an electronic pen in the virtualreality space coordinate system, acquire second information indicating aposition of the electronic pen in the plane coordinate system, andupdate the three-dimensional object in the virtual reality spacecoordinate system based on the first information and update thethree-dimensional object in the virtual reality space coordinate systembased on the second information.
 3. The rendering device according toclaim 2, wherein the instructions, when executed by the processor, causethe processor to: render a tablet terminal image representing a tabletterminal based on a position of the tablet terminal in the virtualreality space coordinate system, wherein the plane coordinate system isdefined based on a position of the tablet terminal in the virtualreality space.
 4. The rendering device according to claim 3, wherein theinstructions, when executed by the processor, cause the processor to:according to a user operation, selectively, display thethree-dimensional object as the three-dimensional object ofthree-dimensional displaying in the virtual reality space coordinatesystem and display the three-dimensional object as the three-dimensionalobject of two-dimensional displaying in the plane coordinate system. 5.The rendering device according to claim 3, wherein the instructions,when executed by the processor, cause the processor to: display theelectronic pen in the tablet terminal image based on the position of theelectronic pen in the virtual reality space coordinate system indicatedby the first information.
 6. The rendering device according to claim 2,wherein the instructions, when executed by the processor, cause theprocessor to: render a display surface image representing a displaysurface on which two-dimensional displaying of a currently-inputthree-dimensional object is carried out, and render a tablet surfaceimage representing a tablet surface based on a position of the tabletsurface in the virtual reality space coordinate system.
 7. The renderingdevice according to claim 6, wherein the display surface image isrendered at a position selected in the virtual reality space or aposition in contact with the three-dimensional object in the virtualreality space.
 8. The rendering device according to claim 6, wherein thetablet surface is not configured to have a function of detecting aposition indicated by the electronic pen.
 9. The rendering deviceaccording to claim 2, wherein a sensing device is disposed for theelectronic pen, and the instructions, when executed by the processor,cause the processor to acquire writing pressure data in response to amanipulation of the sensing device, and update the three-dimensionalobject based on the writing pressure data acquired.
 10. The renderingdevice according to claim 9, wherein the sensing device is a forcesensor that detects a gripping force applied to the electronic pen bythe user, wherein the writing pressure data corresponds to the grippingforce.
 11. A rendering method for rendering a three-dimensional objectin a virtual reality space on a display, the rendering methodcomprising: rendering the three-dimensional object as athree-dimensional object of three-dimensional displaying in a virtualreality space coordinate system; rendering the three-dimensional objectas a three-dimensional object of two-dimensional displaying in a planecoordinate system; and updating displaying of the display based on aresult of the rendering the three-dimensional object as thethree-dimensional object of three-dimensional displaying in the virtualreality space coordinate system and displaying of the display based on aresult of the rendering the three-dimensional object as thethree-dimensional object of two-dimensional displaying in the planecoordinate system.
 12. The rendering method according to claim 11,further comprising: acquiring first information indicating a position ofan electronic pen in the virtual reality space coordinate system,acquiring second information indicating a position of the electronic penin the plane coordinate system, and updating the three-dimensionalobject in the virtual reality space coordinate system based on the firstinformation; and updating the three-dimensional object in the virtualreality space coordinate system based on the second information.
 13. Therendering method according to claim 12, further comprising: rendering atablet terminal image representing a tablet terminal based on a positionof the tablet terminal in the virtual reality space coordinate system,wherein the plane coordinate system is defined based on a position ofthe tablet terminal in the virtual reality space.
 14. The renderingmethod according to claim 13, further comprising: according to a useroperation, selectively displaying the three-dimensional object as thethree-dimensional object of three-dimensional displaying in the virtualreality space coordinate system and displaying the three-dimensionalobject as the three-dimensional object of two-dimensional displaying inthe plane coordinate system.
 15. The rendering method according to claim13, further comprising: displaying the electronic pen in the tabletterminal image based on the position of the electronic pen in thevirtual reality space coordinate system indicated by the firstinformation.
 16. The rendering method according to claim 12, furthercomprising: rendering a display surface image representing a displaysurface on which two-dimensional displaying of a currently-inputthree-dimensional object is carried out, and rendering a tablet surfaceimage representing a tablet surface based on a position of the tabletsurface in the virtual reality space coordinate system.
 17. Therendering method according to claim 16, wherein the rendering thedisplay surface image includes rendering the display surface image at aposition selected in the virtual reality space or a position in contactwith the three-dimensional object in the virtual reality space.
 18. Therendering method according to claim 16, wherein the tablet surface isnot configured to have a function of detecting a position indicated bythe electronic pen.
 19. The rendering method according to claim 12,further comprising: acquiring writing pressure data from a sensingdevice disposed for the electronic pen in response to a manipulation ofthe sensing device, and updating the three-dimensional object based onthe writing pressure data acquired.
 20. The rendering method accordingto claim 19, further comprising: detecting, by the sensing device, agripping force applied to the electronic pen by the user, wherein thewriting pressure data corresponds to the gripping force.